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@ A display system for visualization of body structures during medical procedures. 

@ An Interactive display system superimposes 
radiofogical images on a semi-transparent 
screen through which a surgeon views a patient 
during a medical procedure. The superimposed 
image is derived from image data obtained with 
an imaging system. The radiological image is 
registered with the surgeon's view of the patient 
and displayed in real-time during a medical 
procedure. This allows the surgeon to view 
intemal and external structures and the relation 
between them simultaneously, and adjust the 
procedure accordingly. A second embodiment 
employs stereoscopic viewing methods to pro- 
vide three-dimensional representations of the 
radblogical images superimposed on the 
semi-transparent screen through which the sur- 
geon views the patient. 
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BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a system for aid- 
ing a medical practitioner, such as a surgeon. In vis- 
ualizing anatomical structures during medical proce- 
dures and more specifically to a system which allows 
the practitioner to view, in real time, both Internal and 
external anatomical structures. 

2. Discussion of Prior Art 

This apptication is related to International appli- 
cation US 94/02572 "Computer Graphic and Live Vid- 
eo System for Enhancing Visualization of Body Struc- 
tures During Surgery" and to European Patent appli- 
cations 0549183 "System For Displaying Solid Cuts 
For Surfaces of Solid Models", 0549189 "Solid Mod- 
els Generation By Span Method Using Dividing 
Cubes" and 0549182 "Apparatus and Method For 
Displaying Suiigical Cuts In Three-Dimensional Mod- 
els". 

Presently, during surgery and other medical pro- 
cedures such as endoscopy, biopsy and Implantation, 
physicians view several static radiographic views of 
the patient in the operating room. Typically these are 
transparent film renderings of magnetic resonance 
(MR), computed tomography (CT), conventional X- 
ray images or ultrasound images. Since these im- 
ages are two dimensional static images, the physi- 
cians must determine actual three-dimensional (3D) 
location and shape of desired internal structures 
within the patient from the 2D images which they are 
viewing. Conceptually, the surgeon constructs a 3D 
model of the Internal structures and connelates these 
internal structures with visible external structures of 
the patient where they must cut. This is often difficult 
because the scale and the orientation of the 2D image 
may differ from what the surgeon is seeing, and the 
surgeon may not be able to view both the patient and 
the medical diagnostic Images simultaneously. 

Another technique employed in localization of in- 
ternal structures during surgery is known as stereo- 
tactic surgery as described in "Interactive Stereotac- 
tic Surgical System for the Removal of Intracranial Tu- 
mors Utilizing the CO2 Laser and CT-Derived Data- 
base" by B. A. Kali, P. J. Kelly, and S. J. Goerss. IEEE 
Transactions on Biomedical Engineering, vol. BME- 
32, no. 2. pp 112-116. 1985; and "Comprehensive 
Computer-Assisted Data Collection Treatment Plan- 
ning and Interactive Surgery" by B. A. Kail. P. J Kelly, 
and S. J. Goerss, Medical Imaging, vol. 767 pp. 509- 
514, 1987, With this approach, a rigid mechanical 
frame is attached to the patient before a CT or MR 
procedure. The frame and Its landmarks can be seen 
in tfie resulting images. Mechanisms on the frame 
position a probe at specific location within the Image. 



The disadvantages of this approach are that the 
frame limits access to the patient, and the Images are 
static images which do not follow the patient if he 
moves during surgery. 
5 A third technique used for localization of internal 

structures Is described In "A Frameless Stereotaxic 
Operating Microscope for Neurosurgery" by E. M. 
Friets, J. W. StnDhbehn, J. F. Hatch, and D. W. Ro- 
berts, IEEE Transactions on Biomedical Engineering, 
10 vol. 36., no. 6, pp 608-61 7. June 1 989. 

Three dimensional models of anatomical struc- 
tures can be created from data of different medical 
Imaging nrodalities as described in the applications 
listed above. These applications describe creating 
15 and manipulating models of internal structures of pa- 
tients and providing images of selected structures at 
desired orientations to an operator. These allow visu- 
alization of internal structures as solid models. 
Currently there is a need for a system to aid 
20 physicians in suipery and other medical procedures 
which interactively displays computer generated rep- 
resentations of internal structures in correct relation 
with external structures of the patient 

25 SUMMARY OF THE INVENTION 

A feature of the present invention is to provide a 
system which aids in surgery by simultaneously su- 
perimposing an image of internal structures upon ex- 
30 ternal structures even if the patients changes their 
position or if the operatore change their viewing an- 
gles. 

Another feature of the present Invention is to pro- 
vide an interactive system which displays desired in- 
35 ternal structures upon external structures, having 
tiie same scale and viewed from the same orientation 
angles. 

Another features of the present invention is to 
provide a surgeon a "heads up" display of radiological 
40 images superimposed upon a patient within his field 
of view. 

A real-time surgery apparatus for displaying in- 
teractive internal and externa! Images of a patient 
employs a semi-transparent display screen allowing 

45 an operator to simultaneously view exposed surfaces 
and computer generated images of a patient. 

A medical imaging device obtains three- 
dimensional (3D) imaging data of internal structures 
of the patient which are fed to a workstation. The 

60 workstation creates three-dimensional (3D) comput- 
er generated models which may be manipulated with- 
outfurther need forthe medical Imaging device. Two- 
dimensional computer generated images of the nrrod- 
els are Interactively oriented and scaled so that their 

55 display on a semi-transparent display devfce coin- 
cides with the operator's visual image of the patient. 

The three-dimensional position and orientation of 
the patient, display device and operator's eyes are 
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monitored in real-tinie. This information is then used 
to adapt the computer generated dispiay on the semi- 
transparent screen so that the visual and computer 
generated images consistently coincide. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The features of the invention believed to be novel 
are set forth with particularity in the appended claims. 
The Invention itself, however, both as to organization 
and method of operation, together with further ob- 
jects and advantages thereof, may best be under- 
stood by reference to the following description taken 
in conjunction with the accompanying drawing in 
which: 

Fig. 1 is a simplified block diagram of a first em- 
bodiment of a medksal display apparatus according to 
the present invention. 

Fig. 2 is a simplified block diagram of a second 
embodiment of a medical display apparatus accord- 
ing to the present Invention. 

DETAILED DESCRIPTION OF THE INVENTION 

In Fig. 1, a patient 1 on which a medical proce- 
dure such as surgery is to be perfomned, is scanned 
by a medical Imaging apparatus 10 which may be a 
magnetic resonance (MR) Inr^glng apparatus, a com- 
puted axial tomography (CAT) apparatus, a positron 
emission tomography (PET) or similar imaging device 
capable of creating multi-dlmenstonal volumetric 
data such as 3-dimenslonal (3-D) data, from Internal 
structures of the patient. After imaging, apparatus 10 
provides the volumetric data to a model workstation 
100. Once the volumetric data has been provided to 
model workstation 100, further need for imaging ap- 
paratus 10 Imaging apparatus is no longer required. 
This is important since some medical procedures 
need not be performed with the patient situated within 
the confines of an imaging apparatus, which can be 
constricting in the case of MR imaging. In alternative 
embodiments, imaging apparatus 10 may be Interac- 
tively employed during the medical procedure. Model 
workstation 100 stores the volumetric data and cre- 
ates computer generated models from the data capa- 
ble of being scaled, rotated and otherwise manipulat- 
ed, without the further need for imaging apparatus 
10. 

An operator 350, such as a physician or medical 
assistant monitors patient 1. A semi-transparent 
screen 250 is Interposed between patient 1 and op- 
erator 350. A tracking device 50 which monitors and 
tracks operator 350, semi-transparent screen 250 
and patient 1 determines a relative roll a, pitch 8, and 
yaw ^ orientatton between operator 350, semi- 
transparent screen 250 and subject 1. Tracking de- 
vice 50 may be a 6-degrees of freedom tracking de- 
vice as described The Flock of Birds" Installation and 



Operation Guide, Ascension Technology Corporation, 
July 5. 1992, In the Introduction pp. 1-3, Appendix 1, 
p. 89 and Appendix 3, p. 93. Tracking device 50 also 
determines a location (In Cartesian coordinates) of 

5 operator 350 and semi-transparent screen 250 with 
relation to patient 1. Patient 1 is assumed to be at the 
origin of the Cartesian coordinate system (x,y,z) = 
(0,0,0), therefore all distances relative to the patient 
are simply the (x,y,z) locatfon. The focatfon and ori- 

10 entation are interactively provided to model worksta- 
tion 100 by tracking device 50. The location and ori- 
entation may also be provided manually to model 
workstation(s) in different embodiments. 

Model workstation 100 processes the 3D volu- 

15 metric data it receives and creates selected render- 
ings of the data. One rendering method determines 
surfaces between differing types of tissue. Connec- 
tivity of similar types of tissue adjacentto one another 
is then determined. Differentiation of tissue types 

20 based on the nature of the signal in the three- 
dimensional Image data is known as segmentation. 
When the 3-D volumetric data has been segmented 
Into internal structures, each internal structure may 
be treated as a separate solid object by model work- 

25 station 1 00. The model workstation has the capability 
of selectively displaying desired internal structuies, 
color coding structures and severing, rotating and 
translating internal structures in order to manipulate 
these images in a desired manner to provide vtsuali- 

$0 zation to an operator working model workstation 1 00. 
An alternative rendering method generates two- 
dimensional projections of selected features within 
the three-dimensional data set is described in Euro- 
pean Patent Application 0506302 "PnDjection Meth- 

35 ods for Producing Two-Dimensional Images from 
Three-Dlmensional Data". For example, two-dimen- 
sbnal projection angiograms can be extracted from a 
three-dimensional phase contrast or time-of-ftight 
magnetic resonance angiogram. Several projection 

40 algorithms are possible. These include the detectton 
of the maximum pixel intensity along a selected pro- 
jectk>n ray through the three-dimensional data, deter- 
mination of the average pixel intensity of a selected 
projection ray and the determination of the standard 

45 deviatfon of all pbcels along a selected projection ray. 
Model workstation 1 00 receives input data from 
a model cut plane input device 40 and a workstation 
view input device 60 to select the method of display- 
ing internal structures of patient 1 . Model cut plane in- 

50 put device 40 and a workstation view input device 60 
may be a computer pointing device such as a mouse 
or trackball, or any input device which indicates 
planes In which to cut the images and a viewing angle 
and scale. Tracking device 50 provides relative orlen- 

55 tation data between operator 350 and patient 1 and 
scaling, which allows model workstation 100 to syn- 
thesize an interactive computer generated image of 
Internal structures of patient 1 and have it coincide 
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with the real-time scene viewed by operator 350. This 
Interactive computer generated image may t>e con- 
verted from a computer monitor signal, such as an 
RGB computer monitor signal, to a video format by 
passing It through a scan converter 192 (shown in 5 
phantom), depending on the display format of semi- 
transparent screen 250. The computer generated im- 
age is provided to semi-transparent screen 250 which 
is interposed between patient 1 and operator 350. 
Semi-transparent screen 250 provides a desired mix- io 
ture of external structures of patient 1 seen by oper- 
ator 350 and the computer generated image from 
model workstation 100. Semi-transparent screen 250 
may receive input signals from the operator, for exam- 
ple, through workstation view input 60. This may in- i$ 
volve the degree of transparency of each image, or 
any of various other special effects. One very useful 
special effect is a movable window which has 0% 
transparency (100% opaqueness) superimposed 
upon another Image. When the window image is of In- 20 
ternal structures superimposed upon external struc- 
tures, it creates the iliuston of external structures cut 
away within the window, exposing underlying internal 
structures. Other conventional video special effects 
may also be employed. Semi-transparent screen 250 25 
allows operator 350 to visualize both internal and ex- 
ternal structures simultaneously. The resulting scene 
witnessed by operator 350 is an Interactive real-time 
image of patient 1, even if the patient moves during 
the medical procedure. Since internal structures and 30 
their relation to exposed external structures are si- 
multaneously displayed, a surgeon using the present 
invention will perceive a very accurate indication of 
where he should cut through external structures to ar- 
rive at a desired internal structure while avoiding vital 35 
internal structures. 

Figure 2 illustrates an alternative embodiment of 
the present invention in which operator 350 receives 
a stereoscopic view of internal and external struc- 
tures of patient 1. Each of the eyes of operator 350 4o 
differe In its orientation with relation to the patient 
therefore a different view is provided to each eye. 
Tracking device 50 tracks the relative location 
(xi,yi,zi) and orientation angle (at, 9i) between a 
first eye of operator 350 and patient 1 . Tracking de- 45 
vice 50 also tracl« a second location (X2,y2.22) and 
orientation (a2,fc»©2) between a second eye of oper- 
ator 350 and patient 1. The second location and ori- 
entation can be measured Independently by tracking 
device 50; or the location and orientation of the first so 
eye can be used to compute the location and orienta- 
tion of the second eye. The locations and orientations 
are fed to a first model workstation 100a, and a sec- 
ond model workstation 100b which create a right and 
left computer graphic image at locations (x,,y,,zO, 55 
(X2,y2,z2)» respectively, and orientations (a,, 61) 
(a2»*2,92)» respectively, corresponding to the views of 
each eye of operator 350, respectively. Control sig- 



nals from model cut plane 40, workstatton view input 
60 and Imaging apparatus 1 0 can be sent to both first 
model workstation 1 00a, which in turn propagates the 
control signals to second model workstation 100b via 
communications path 140, or alternatively, the con- 
trol signals can be sent directly to both model work- 
stations directly. 

The left and right computer generated image, 
pertaining to a left and right view, respectively, are 
converted to video fonmat if required by semi- 
transparent screen 250. This Is accomplished by scan 
converters 192a, 192b (shown In phantom) which 
pass the converted computer generated signals to a 
sequencer 198. Sequencer 198 may be a convention- 
al video sequencer as described in Tortable, Low 
Cost Devices for Videotaping, Editing and Displaying 
Field Sequential Stereoscopic Motion Pictures and 
Video" by M. Starics, Stereoscopic Displays and Ap- 
plications Proc. SPIE Vol. 1256, pp. 266-271, 1990. 

Sequencer 198 passes the left computer gener- 
ated Image to semi-transparent screen 250. Se- 
quencer 1 98 then passes the right computer generat- 
ed image to semi-transparent screen 250. Sequencer 
198 alternates many times per second, in synchroni- 
zation, between right and left views. 

The image displayed on semi-transparent screen 
250 is time multiplexed to produce an image to the left 
eye and right eye of the operator in an alternating 
fashion. A stereoscopic viewer 262 is synchronized to 
a sequencer 198 and operates to block the vision of 
the operator's left or right eye allowing the opposite 
eye to view the image on semi-transparent screen 
250 for an instant and vice-verea. This allows opera- 
tor 350 to see the left image with the left eye while the 
right eye sees nothing and the right image with the 
right eye while the left eye sees nothing in rapid suc- 
cession. This creates a stereoscopic illusion, adding 
the dimension of depth perception in viewing the im- 
age displayed on semi-transparent screen 250. 
Depth perception is very valuable in surgery since it 
adds a dimension that assists in visually localizing 
structures, which is especially important in complex, 
delicate surgery. 

For both embodiments of the invention the com- 
puter image must be registered (coincide) with the ex- 
ternal structures as viewed by operator 350. Initiali- 
zation may be accomplished by manual inputfrom the 
operator to rotate, translate and scale the computer 
generated image{s) until they coincide with the scene 
observed through the semi-transparent screen, or by 
employing tracking device 50 to set initial parameters. 

Once the 3D model and the visual image of pa- 
tient 1 are aligned, tracking device 50 keeps the view 
angles and-fleld of view consistent This allows real- 
time Interactive synchronization between the opera- 
tor's view of patient 1 and the computer generated im- 
age(s). 

Since the image of each internal structure can be 
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segmented into what resembles a solid object, it may 
be manipulated as a solid object. In the case of struc- 
tures of a patient, a surgeon may acquire data from 
the patient by medical Imaging, then plan surgery by 
manipulating the models to plan a desired result be- 5 
fore surgery. This is common in complex reconstruc- 
tive surgery. Once the plan is determined, it may be 
stored and played back during surgery. The Images 
of internal structures are interactively oriented and 
scaled to coincide with the actual patient. 10 

A user employs as a model cut piane input device, 
a workstation view input device (40, 60 of Figs. 1 and 
2, respectively) to select planes in which to cut the 
structures In the model, to select a three-dimensional 
orientation of the model, and to select screen cut 15 
planes which define a workstation viewing region. 
The model workstatton can incorporate a dipping cir- 
cuit to determine points within the model cut planes, 
a rotation circuit to rotate points and normal vectors, 
a segmentation processor, and a shading circuit 20 
which determines shading based upon the orientation 
of the normal vector at each point, in addition a screen 
clipping circuit can be used to determine points within 
a region defined by the screen cut planes. The model 
workstation also can Include a display circuit to create 25 
video signals which, when propagated to a suitable 
display device, generate images of multiple surfaces 
that are within the deslrad display region and the 
screen cut planes. 

in radical surgery such as ablative surgery, or 30 
massive trauma cases, there is little structure which 
remains to correctly determine what a normal anato- 
my should be. In these cases, an additional model 
workstation may have a model of normal structures 
stored which may be mixed with the other i mages be- 35 
Ing displayed to act as a guide in reconstructive sur- 
gery. This may be implemented by additional work- 
stations or model manipulation boards. 

in the present embodiments of the invention, 
semi-transparent screen 250 is Interposed between 40 
operator 350 and patient 1. Screen 250 can be con- 
structed with a liquid crystal display or it can be com- 
prised of a partially slivered mirror reflecting an image 
from a video monitor. Semi-transparent display 250 
can be constructed as a relatively large device having 45 
dimensions approximately equal to that of the region 
of interest of patient 1, or alternatively it can be of 
small dimension and placed relatively close to the op- 
erator's eyes, perhaps incorporated into headgear or 
eyewear. so 

While several presently preferred embodiments 
of the novel visualization system have been descri- 
bed in detail herein, many modifications and varia- 
tions will now become apparent to those skilled in the 
art. It is, therefore, to be understood that the append- 55 
ed claims are intended to cover all such modif k:ations 
and variations 



Claims 

1 . A reai-ti me medical apparatus for displaying to an 
operator interactive images of internal structures 
of a patient coinciding with a view by the operator 
of the patient comprising: 

a) a medical imaging system for obtaining 
multi-dimensional imaging data of internal 
structures of sakJ patient; 

b) a workstation for creating Images from the 
inrtaging data of internal structures of said pa- 
tient viewed from a location (x,y,z) and orien- 
tation (a,<|>,e) which coincide with said view of 
the patient by said operator; 

c) a semi-transparent screen adapted for al- 
lowing said operator to view said patient 
through the semi-transparent screen and 
adapted for displaying the images of internal 
structures from the workstation with a desired 
degree of transparency to create the illusion 
of internal structures superimposed upon 
saki patient 

2. The real-time medical apparatus of claim 1 fur- 
ther comprising a tracking device adapted to 
measure the location (x,y,z) and orientation 
(a,^,0) of said operator with respect to sakI pa- 
tient, for repeatedly providing these measure- 
ments to the workstation. 

3. The real-time medical apparatus of daim 1 
wherein the workstatton further comprises surgi- 
cal planning means adapted to create models of 
internal structures of said patient, interactively 
manipulate the models to result in modifications 
to internal structures, store the models and mod- 
ifications and display the models and nrtodif ica- 
tions viewed from a location and orientation to co- 
incide with the view of the patient by the operator. 

4. The real-time medical apparatus of daim 1 
wherein the workstation incorporates normal 
anatomical models of internal structures, and is 
adapted to display the models viewed from a lo- 
cation and orientation to coindde with the view of 
the patient by the operator, such that sakJ work- 
station can act as a guide in reconstructive sur- 
gery. 

5. Areal-time medical apparatus for displaying to an 
operator interactive three-dimensional (3D) inter- 
nal and external images of a patient comprising: 

a) a medical imaging system for obtaining 
three-dimensional (3D) imaging data of inter- 
nal structures of sakl patient; 

b) tracking means for measuring locations 
(xi,yi,zi) (x2,y2iZ2) and orientation angles 
(ai,<|)i,ei) (a2,<^.02) of each eye of said opera- 
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tor with respect to said patient; 

c) a first workstation for creating a right gohd- 
puter generated image of internal structures 
of said patient from the 3D imaging data from 

the medical imaging system as viewed from s 
the location (xi,yi,zi) and orientation angle 
(ai><^i.Oi) obtained from the tracking means; 

d) a second workstation for creating a left 
computer generated image of internal struc- 
tures of said patient from the 3D imaging data io 
from the medicai imaging system as viewed 
from a location (x2,y2,Z2) and orientation angle 
(a2,fc,02) obtained from the tracking means; 

e) a semi-transparent screen adapted to allow 
said operator to view said patient through the i$ 
semi-transparent screen and adapted to dis- 
play the left and right computer generated im- 
ages of internal structures from the worksta- 
tions with a desired degree of transparency in 
rapid succession; 20 
0 stereoscopic display means synchronized 

with the semi-transparent screen for allowing 
only the operator's right eye to view the semi- 
transparent screen when the right computer 
generated Image is being displayed, and for 2S 
allowing only the operator's left eye to view 
the semMransparent screen when the left 
computer generated Inrage Is being displayed 
thereby simulating a 3D inrmge of internal and 
external structures of said patient 30 

6. A method of aiding an operator to perform a med- 
ical procedure upon a patient comprising the 
steps of: 

a) acquiring multi-dimensional medical imag- 35 
ing data from Internal structures of said pa- 
tient; 

b) measuring a location (x.y.z) and orientation 
angle (a.(|>,e) of said patient as viewed by said 
operator; 40 

c) creating a computer generated image of the 
internal structures from the medical imaging 
data viewed from the locatkMi (x,y,z) and ori- 
entation angle (a,<|),6); 

d) interposing a semi-transparent screen be- 4S 
tween said operator and said patient aik)wing 

said operator to view said patient through the 
semi-transparent screen; and 

e) displaying the computer generated image 

on the semi-transparent screen with a desired so 
degree of transparency to create the illusion 
of internal structures superimposed upon 
said patient to assist the operator in said med- 
ical procedure. 



a medical imaging system. 

8. The nf^ethod of aiding an operator in a medical 
procedure of claim 6 further comprising the step 
of creating a computer generated gukie image of 
nonfnal anatomical Internal structures viewed 
from a location (x,y,z) and orientation angle 
(a,<l>,0) to be used as a guide to reconstructive 
surgery; and mbcing portions of the guide Image 
with the computer generated images. 

9. A method of aiding an operator In a medical pro- 
cedure of a patient comprising the steps of: 

a) obtaining three-dimensional (3D) imaging 
data of internal structures of saM patient; 

b) interposing a semi-transparent screen be- 
tween said operator and said patient allowing 
said operator to view said patient through the 
semi-transparent screen; 

c) measuring the iocatfons (x,.yi,zi) (X2.y2.z2) 
and orientation angles (ai,(|>,.ei) {a2,h^^ be- 
tween said patient and each eye of said oper- 
ator, respectively, 

d) creating a left computer generated image 
and a right computer generated image of In- 
ternal structures from the 3D imaging data of 
said patient viewed from distances (xi,yi,zi) 
(X2.y2.z2) and orientation angles (ai,*,,ei) 
(a2.*2,^ respectively; 

e) displaying the left and right computer gen- 
erated Images of Internal structures of a de- 
sired degree of transparency In rapid succes- 
sion on the semi-transparent screen; and 

f) blocking the view of the left eye of the op- 
erator when the right computer generated Im- 
age is being displayed, and blocking the view 
of the right eye of the operator when the left 
computer generated image is being displayed 
thereby simulating a 3D Image of internal and 
external structures of said patient 

10. The method of aiding an operator in a medical 
procedure of claim 9 further comprising the step 
of adjusting the distances (xi,yi,zi) (X2,y2,z2) and 
orientation angles {ai,4>i,ei) (a2,<k,92) of the left 
and right computer generated images to corre- 
spond to the view of the patient by the operator. 



The method of aiding an operator in a medical 
procedure of claim 6 wherein the step of acquiring 
medical imaging data is performed in real-time by 
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